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LONG-TERM  GOALS 


The  long-term  goal  of  this  research  is  to  develop  an  efficient,  robust  hydrodynamic  model  that  is 
eapable  of  three-dimensional  (3D),  prognostic  baroclinic  simulations  on  a  parallel  eomputing  platform. 
Our  approach  is  to  modify  an  existing  model  ADCIRC  (ADvanced  3D  ClRCulation  model  [1]),  which 
is  the  product  of  over  20  years  of  researeh  and  development  by  scientists  and  engineers  at  six  different 
institutions.  Among  other  military  and  civilian  applications,  ADCIRC  is  currently  one  of  a  suite  of 
models  used  by  the  Offiee  of  Naval  Research’s  (ONR)  Naval  Research  Lab  (NRL)  at  Stennis  Space 
Center,  MS,  to  help  plan  Naval  fleet  operations  in  ocean  basins  around  the  world.  The  improved 
ADCIRC  model  would  further  enhanee  NRL’s  ability  to  provide  aceurate,  timely  nowcasts/foreeasts, 
partieularly  in  thermohaline  driven  basins,  like  the  Persian  Gulf  (see  Figure  1). 


Figure  1.  Diagnostic  density  field  (bottom  layer)  for  the  Persian  Gulf.  Note  that  the  large  density 
gradients  produce  a  large  baroclinic  pressure  gradient  field  that  can  result  in  currents  that  equal  or 

exceed  those  generated  by  the  tides  or  winds. 
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OBJECTIVES 


Computer  modeling  of  highly  eomplex,  nonlinear,  dynamical  systems,  such  as  that  shown  in  Figure  1, 
brings  together  many  scientific  disciplines,  including  in  our  case,  physical  oceanography,  meteorology, 
fluid  dynamics,  numerical  methods,  and  computer  science.  Since,  by  definition,  a  model  is  a  simplified 
portrayal  of  reality,  successful  model  applications  depend  on  how  that  simplification  is  realized.  Thus, 
our  primary  research  objective  is  to  find  the  balance  between  the  scale  of  physics  to  include  in  the 
model,  the  type  of  numerical  algorithm,  the  amount  of  resolution,  and  the  computational  efficiency  on 
parallel  platforms  in  order  to  obtain  accurate,  robust,  and  timely  simulations. 

APPROACH 

As  presented  in  the  original  proposal,  our  specific  approach  to  meeting  the  proposed  objective  is  to 
study  and  advance  four  aspects  of  the  current  ADCIRC  model:  1)  compare  several  transport  algorithms 
and  program  the  most  promising  one  into  a  transport  code  for  prognostic  baroclinic  simulations;  2) 
understand  the  interplay  between  horizontal  and  vertical  resolution  for  3D  simulations  for  both 
diagnostic  and  prognostic  simulations;  3)  define  a  criteria  for  switching  between  the  finite  element  and 
discontinuous  Galerkin  solution  algorithms;  and  4)  program  the  3D  version  of  the  code  to  run  in  a 
distributed  memory,  parallel  computing  environment.  In  addition,  we  have  undertaken  two 
complementary  tasks  that  were  not  part  of  the  original  scope  of  work:  5)  look  at  alternative  I/O 
paradigms  to  reduce  output  file  sizes  and  speed  up  throughput;  6)  develop  a  web-based  grid  library  to 
archive  the  many  computational  grids  that  have  been,  and  will  be,  developed  by  ADCIRC  researchers. 
Key  individuals  and  their  roles  are  as  follows  (all  are  at  the  University  of  Oklahoma,  unless  noted 
otherwise):  R.  Kolar  (PI),  project  oversight;  J.  Atkinson  (post-doc),  transport  model  development;  K. 
Dresback  (Ph.D.  student),  resolution  studies  and  algorithms  to  compute  the  baroclinic  pressure 
gradient;  C.  Szpilka  (Ph.D.  student)  and  C.  Dawson  (U.  Texas),  transport  model  development  and 
discontinuous  Galerkin/continuous  Galerkin  algorithm  coupling;  J.  Dietrich  (undergraduate 
researcher),  numerical  experiments;  P.  Quarles  (M.S.  student),  grid  data  base;  L.  McSpadden 
(undergraduate  researcher),  visualization  and  I/O;  J.  Antonio,  S.  Dhall,  S.  Lakshmivarahan  (Co-PI’s), 
parallel  programming. 

WORK  COMPLETED 

Task  1  -  Prognostic  Baroclinic  Simulations.  These  simulations  require  that  the  hydrodynamic  model 
be  dynamically  linked  to  a  transport  model  that  has  been  fitted  with  an  appropriate  equation-of-state.  It 
became  evident  during  the  early  stages  of  this  project  that  in  order  to  test  a  variety  of  transport 
algorithms  in  a  timely  fashion,  we  needed  to  develop  a  2D  laterally  averaged  (“x-z”)  model,  which 
takes  about  an  order  of  magnitude  less  time  to  run  than  a  comparable  full  3D  simulation.  Moreover, 
analytical  solutions  exist  to  validate  the  algorithm  in  this  reduced  domain.  Thus,  the  first  year  of  the 
project  was  spent  developing  and  verifying  the  x-z  hydrodynamic  model,  which  we  have  recently 
completed.  Simultaneous  with  the  x-z  hydrodynamic  code  development,  we  spent  time  surveying  the 
literature  as  to  what  other  research  groups  have  done  with  respect  to  the  baroclinic  transport.  The 
results  of  that  survey  are  guiding  our  development  of  the  transport  code.  To  serve  as  a  baseline  for 
comparison,  we  initially  discretized  the  transport  equation  using  a  continuous  Galerkin  algorithm 
(same  as  that  used  for  the  hydrodynamics).  However,  literature  and  experience  suggest  that  shock- 
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capturing  algorithms  (so-called  TVD  algorithms)  may  be  a  better  fit  for  our  intended  applications,  so 
we  have  also  begun  eoding  a  transport  eode  based  on  the  discontinuous  Galerkin  method  [2]. 

Task  2  -  Resolution  Studies.  Sinee  the  prognostie  code  is  still  under  development,  these  resolution 
studies  have  not  begun.  However,  the  current  version  of  ADCIRC  does  have  diagnostie  capabilities, 
but  they  have  not  been  rigorously  tested  on  real  basins.  Consequently,  as  a  first  step  toward  “real” 
simulations,  we  tried  simulating  the  Persian  Gulf,  which  is  known  to  have  significant  contributions  to 
the  flow  field  from  the  buoyancy  terms  (see  Figure  1).  The  computational  grid  for  this  work  is  shown 
in  Figure  2  below.  To  obtain  meaningful  results,  three  modifications  were  needed,  as  discussed  in  the 
Results  section  below. 


Figure  2.  Computational  mesh  and  bathymetry  (darker  blue  for  deeper  bathymetry)  for  the  Persian 
Gulf  domain.  Note  that  the  use  of  triangular  elements  allows  one  to  map  irregular  coastlines 

accurately  and  use  graded  meshes. 

Task  3  -  Algorithm  Coupling.  A  truism  in  numerical  methods  is  that  there  is  no  “best”  algorithm, 
which  also  applies  to  the  ADCIRC  model.  The  Galerkin  finite  element,  GWC  (Generalized  Wave 
Continuity)  algorithm,  whieh  eurrently  drives  the  ADCIRC  model,  is  very  effieient  and  accurate  for 
slowly-evolving,  deep  water  fields,  but  suffers  from  instabilities  in  highly  adveetive  regions.  Thus, 
much  effort  has  been  spent  over  the  last  1 0  years  to  search  for  a  more  stable  algorithm;  the 
discontinuous  Galerkin  (DG)  method  has  emerged  as  the  likely  candidate.  However,  it  is  mueh  less 
computational  efficient  than  the  GWC.  So,  an  optimal  solution  for  us  appears  to  be  to  use  the  GWC  in 
deep  water  and  switch  to  the  DG  in  shallow,  near-shore  regions  [3].  Such  a  multi-algorithmic  code 
raises  new  research  issues,  viz,  how  to  couple  the  two  algorithms  and  where  in  the  domain  to  switch 
from  one  algorithm  to  the  other.  Our  research  group  has  developed  a  numerieal  analysis  tool  that 
quantifies  the  ability  of  an  algorithm  to  propagate  all  possible  wave  frequencies,  ranging  from  high 
frequency  noise  to  long  wave  tidal  motions  [4].  We  are  applying  this  tool  to  first  the  DG  algorithm, 
and  then  the  coupled  DG/GWC  algorithm  in  order  to  verily  that  the  coupling  algorithm  is  indeed 
passing  waves  without  distortion,  and  we  are  using  the  dispersion  characteristics  of  the  algorithm  to 
develop  criteria  for  switching  between  the  two.  Methods  of  coupling  range  from  passing  fluxes  across 
interface  boundaries  to  using  an  overlapping  layer  of  elements  at  the  interface. 
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Task  4  -  Parallel  Computing.  Previous  grants  have  supported  efforts  to  parallelize  the  2D  version  of 
ADCIRC  using  MPI  and  domain  decomposition  so  that  it  runs  on  massively  parallel  clusters.  During 
the  past  year,  we  have  run  extensive  benchmarking  tests  on  two  different  16-processor  platforms,  and 
we  have  begun  benchmarking  efforts  on  a  larger  256  node  parallel  cluster.  3D  parallel  coding  and 
benchmarking  has  not  progressed  much  during  the  past  year. 

Task  5  -  File  I/O.  Full  3D  simulations  produce  enormous  volumes  of  output,  which  can  be  on  the 
order  of  gigabytes.  Consequently,  an  I/O  strategy  must  be  employed  that  is  more  efficient  than  the 
current  ADCIRC  standard,  which  is  ASCII.  Our  team  modified  the  ADCIRC  code  so  that  NetCDF 
(Network  Common  Data  Form)  can  be  used  as  an  output  option. 

Task  6  -  Grid  Database.  Currently,  ADCIRC  grids  reside  with  the  person(s)  who  developed  the  grid; 
when  they  are  shared,  little  or  no  documentation  is  provided.  During  the  past  year,  we  have  developed 
a  web-based  prototype  for  uploading,  archiving,  and  retrieving  ADCIRC  finite  element  meshes.  The 
archived  files  will  contain  all  pertinent  attributes  for  each  grid,  e.g.,  resolution,  source  of  coastline  and 
bathymetric  data,  software  used  to  generate  the  mesh,  meshing  criteria,  intended  application  for  the 
grid,  and  security  classification. 

RESULTS 

Task  1 .  An  explicit  CG  transport  requires  such  small  time  steps  for  stability,  that  it  is  not  a  viable 
algorithm  for  full  3D  transport.  Moreover,  the  algorithm  is  not  stable  at  any  time  step  for  sharp  front 
problems,  such  as  a  “dam  break”. 

Task  2.  Extensive  testing,  analysis,  and  debugging  of  ADCIRC  led  to  the  following  modifications  to 
the  3D  diagnostic  code:  a  better  algorithm  for  computing  baroclinic  pressure  gradients  when  neighbor 
nodes  “run  into  the  ground”  (occurs  in  areas  of  steep  bathymetric  gradients);  a  different  criteria  for 
checking  whether  or  not  a  vertical  node  is  at  or  near  the  sea  bed;  a  more  robust  procedure  for  using  the 
OAX  interpolating  code  [5,  6]  to  project  the  temperature  and  salinity  field  (data  obtained  from 
MODAS)  onto  the  computational  grid.  Once  these  modifications  were  made,  we  were  able  to  obtain 
realistic  baroclinic  simulations  for  the  Persian  Gulf,  as  shown  in  Figure  3  below. 
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Figure  3.  3D  diagnostic  simulation  results  for  the  Persian  Gulf  using  the  density  field  shown  in 
Figure  1  (expanded  view  of  the  center  of  the  domain), 

Task3.  Work  in  this  area  is  on-going,  but  two  significant  observations  stand  out.  First,  the 
aforementioned  phase  analysis  tool  has  identified  that  partial  coupling  (e.g.,  applying  DG  to  just  the 
continuity  equation)  can  introduce  artificial  oscillations  in  the  solution  unless  slope  limiters  are 
applied,  as  shown  in  Figure  4  below.  Recent  applications  have  supported  this  theoretical  result.  Full 
DG/CG  coupling  through  overlapping  layers  of  elements  appears  to  be  promising  (preserves 
amplitudes  and  phases  across  the  interface),  plus  it  follows  the  same  paradigm  that  we  use  for  domain 
decomposition  [7]. 
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ver2b:  piecewise  linear  Legendre  DG  linear  Lagrange  CG  -  boundary  forcing 
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Figure  4.  Phase  analysis  of partial  DG/CG  coupling:  upper  left  and  right  frames  show  damping 
ratio  and  phase  error  vs.  wavelength,  respectively;  lower  left  frame  shows  dispersion  relation;  lower 
right  frame  shows  simulation  result  for  75dx  wave.  Note  the  oscillations  in  the  solution,  as  indicated 

by  the  black  dots,  for  high  frequency  waves. 

Task  4.  Benchmarking  results  indicate  that  the  2D  code  scales  well  on  both  16  processor  clusters,  as 
shown  in  Figure  5,  provided  the  problem  size  is  large  enough  so  that  inter-processor  communication 
does  not  dominate.  Thus,  the  same  paradigm  (domain  decomposition)  will  be  used  for  the  baroclinic 
3D  code. 
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Figure  5.  Benchmarking  (wall-clock  time  vs.  number  of  processors)  of  the  2D  barotropic  ADCIRC 
model  on  a  16-processor  Sun  Netra  XI  cluster  (navy  line)  and  a  16-processor  P3  Linux  cluster  (red 

line).  Both  show  near  theoretical  speedup. 

Task  5.  In  all  test  cases,  the  use  of  NetCDF  reduced  fde  size  by  2/3  (as  compared  to  ASCII),  and,  for 
I/O  intensive  simulations,  the  cpu  time  was  cut  by  1/2  [8]. 

Task  6.  No  significant  results  to  report. 

IMPACT/APPLICATIONS 

Nothing  to  report  at  this  time  (first  year  of  the  project). 

RELATED  PROJECTS 

With  support  from  the  US  Army  Corps  of  Engineers,  Waterways  Experiment  Station,  colleagues  at  the 
University  of  Notre  Dame  are  working  on  a  sediment  transport  algorithm  that  can  be  coupled  to  the 
ADCIRC  model.  This  modification  of  the  code  is  needed  to  be  able  to  simulate  the  effects  of  coastal 
dredging  and,  thus,  better  manage  coastal  waterways.  Sediment  transport  shares  some  of  the  same 
issues  as  salinity  and  temperature  transport  that  we  are  working  on;  consequently,  we  are  sharing 
research  results,  particularly  since  both  groups  are  looking  at  similar  transport  algorithms  (e.g., 
discontinuous  Galerkin). 
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